Conformally invariant trilinear forms on the sphere 

Jean-Louis Clerc and Bent 0rsted 
January 4, 2010 

Abstract 

To each complex number A is associated a representation tta of 
the conformal group SOo{l,n) on C°°(S'"~^) (spherical principal se- 
ries). For three values Ai,A2,A3, we construct a trilinear form on 
C°°(S'"-i) X C°°(S'"-i) X C°°(S'"-i), which is invariant by ttx, ® tta, ® 
TTAg. The trilinear form, first defined for (Ai,A2,A3) in an open set of 
C'^ is extended meromorphically, with simple poles located in an ex- 
plicit family of hyperplanes. For generic values of the parameters, we 
prove uniqueness of trilinear invariant forms. 

°2000 Mathematics Subject Classification : 22E45, 43A85 

Introduction 

The motivation for this article came from the paper by J. Bernstein and 
A. Reznikov. In order to estimate automorphic coefficients, they use trilinear 
invariant forms for G = PGL2(M). Their paper shows the importance of 
three related questions of harmonic analysis on PGL2 (M) : 

given three representations {iti,T-Ii), (1^2,^-12), {'^Sj'Hs) in the principal 
sphericaQ series of the group G, 

i) construct a trilinear invariant functional on ^1 x ^2 x ^^3 

ii) prove uniqueness (up to a scalar) of such a trilinear invariant func- 
tional 

in) compute the value of the trilinear functional on the K fixed vectors 
of T-L\,'H2,'Hz respectively. 

The representations are realized on the unit circle, on which the group 
G acts projectively, and indexed by a complex number. One possible gener- 
alization consists in replacing the unit circle by the n — 1 dimensional sphere 
5, under the action of the conformal group G = SOo{l,n). For this case, 

^with respect to the maximal compact subgroup K = PO{2) of G 
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we present here a construction of an invariant trilinear form (item i)), which 
uses an analytic continuation over three complex parameters and discuss the 
uniqueness statement (item ii)) for generic values of the parameters. The 
computation of the normalization factor (item iii)), even for more geometric 
situations, will be published elsewhere, (cf [3]). 

In section 1, we recall elementary facts about conformal geometry of the 
sphere (in particular we give a description of the orbits of G in S* x S* x 5) , 
and introduce the representations which the paper is concerned with. In 
section 2, we define formally the trilinear invariant form, study the domain 
of convergence of the corresponding integral and determine the analytic 
continuation in the three complex parameters corresponding to the three 
representations. In section 3, we prove the uniqueness statement. The proof 
relies on Bruhat's theory, which we recall in an appendix at the end of the 
paper. In section 4, we give an alternative approach to the construction of 
an invariant trilinear form, using a realization of the tensor product of two 
of the representations involved, thus making connection with (llj . 

The present paper only deals with regular values of the parameters. 
The residues at poles will yield new conformally invariant trilinear forms, 
supported by the singular orbits oi G in S x S x S and involving differential 
operators akin the Yamabe operator on the sphere, worth of a further study. 
Other geometric situations are potential domains for similar results. Let 
Pi,P2,P3 be three parabolic subgroups of a semi-simple Lie group, such 
that G acts on G/Pi x G/P2 x GjPj, with a finite number of orbits (see 
[1] ) [ID] ) • Take three representations induced by characters of Pi , P2 j -P3 • 
Invariant trilinear forms for these three representations can plausibly be 
studied along the same lines as in the present paper. The case of three 
copies of the Shilov boundary 5 of a bounded symmetric domain of tube 
type is specially appealing (see [3] for a description of the orbits of G in 
SxSxS). 

Let us mention the paper by A. Deitmar [5j, which has some overlap 
with our results. Previous work on the subject also includes [12] and [9]. 

The first author thanks D. Barlet and L. Berard Bergery for conversa- 
tions on various aspects of this paper and the Mathematics Department of 
Aarhus University for welcoming him during the preparation of the present 
work. 
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1 Conformal geometry of the sphere 

Let 

S = S'^-l = {x = {xi,X2, . . . , Xn), \xf =xl + xl + ---+xl = l} 

be the unit sphere in R". We usually (and tacitly) assume n > 3, as the 
case n = 2 needs a few minor changes, which are occasionally mentioned in 
the text. The group K = SO{n) operates on S. Let 

1+ = (1,0,...,0), i- = (-i,o,...,o). 

The stabilizer of 1"*" in K is the subgroup 

M^SO(n-l) = |(j °),uG50(n-l)| . 

With this notation, S ~ K/M is a compact Riemannian symmetric space. 

Another realization of the sphere is useful. Let M^'" be the real vector 
space W^'^^ with the quadratic form 

q{x) = [x,x\= xl- {x{+ xl-\ Vxl). (1) 

To X = {xi,X2-, . . . , Xn) in S associate x = (1, xi,X2, ■ ■ ■ , Xn) in R^'". The 
correspondance 

X \ — > Rx 

associates to a point in S an isotropic line in R^'". The correspondance is 

easily seen to be bijective. The group G = SOQ{l,n) acts naturally on the 
set of isotropic lines, and hence on S. Explicitly, for x in S and g in G, g{x) 
is the unique point in S such that 

g{x) = {gx)o~^ gx . 

For X in S and g in G, set 

'^{9,x) = {gx)o~'^ . (2) 
Let x,y he in S. The following identity holds 

[x,y\ = l- {x,y) = ^\x-yf (3) 

so that for g in G, 

\9{x) -9{y)\ = ii{9,x)^ \x-y\ K{g,y)^ ■ (4) 
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The infinitesimal version of dH is 



\Dg{x)C\ = K{g,xm 



(5) 



for ^ any tangent vector to S at x, where g is in G and Dg{x) stands for 
the differential at x of the map x i— >• g{x). Hence the action of G on 5 is 
conformal, and K{g, x) is interpreted as the conformal factor of g at x. 

We look at as a subgroup of G. It is a maximal compact subgroup of 
G. The stabilizer in G of the point 1+ is the parabolic subgroup P = MAN , 
where 
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The element (t G M) acts on 5 by 
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Let N be the subgroup image of N by the standard Cartan involution 
oiG{g^{gT^) : 
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The map 



c:C^n^(l+) 



V 



is a diffcomorphism from onto \ {1~}. Its inverse is the classical 

stereographic projection from the source 1~ onto the tangent space T^+S to 
S at When using this chart on S, we refer to the noncompact picture. 

Proposition 1.1. The conformal factor K{g,x) is a smooth function of both 
g and x, which satisfies the following properties : 

i) ^91,92 eG,x e S, 

k{9i92,x) = K{gi,g2{x)) K{g2,x) (6) 

a) ygeG,x e S K{g,g-^{x)) = K{g-'^,x)-^ 

Hi) \/x G S,k & K K{k, x) = 1 

iv) \/x e S,t eM., K{at, x) = (cosh t + xi sinh t)~^ . 

Let g in G. As the dimension of the tangent space TxS is n — 1, the 
Jacobian of ^ at x is given by 



j{9,x) = K{g,x) 



ra-l 



(7) 



The map c : ^ — S" \ { — 1} is also conformal. In fact, one has the 
following relation, valid for any ^, ?7 G M"~^ : 



m-c{v)\ = 



m-v\ 



(l + |CP)2(l + |r?|2)2 



(8) 



and its infinitesimal version 



\dciC)C\ 



ICI 



1 + 

{( G M"~^). The corresponding integration formula reads 

2n-l 



/ f{x)da{x) = [ 



(9) 



(10) 



Later, we will need a description of the orbits of G in S x S x S (for the 
diagonal action of G). Recall first that the group G in its diagonal action 
on S X S has two orbits : 
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= {{x,y) eSxS,x^y}, A5 = {{x,x),x e S}. 

As base-point in S^, choose (1+, The stabilizer of (1+, 1~) in G is 
the subgroup MA. 

Proposition 1.2. Let n > 3. There are 5 orbits ofGinSxSxS, namely 

Oq = {{xi,X2,Xs),Xi 7^ Xj for i / j} 

01 = {{xi,x,x),x / Xl} 

02 = {{X,X2,X),X ^ X2} 

03 = {{X,X,X3),X 7^ X3} 

04 = {{x,x,x)} . 

Proof. The five subsets of S x S x S are invariant under the diagonal action 
of G. So it suffices to show that G is transitive on each of these sets. 

• G is transitive on S, hence on O4. Choose (l"*", 1+, 1"'") as base-point. 
The stabilizer of the base-point in G is the subgroup P = MAN. 

• The stabilizer P of 1+ is transitive on 5 \ {1+} (the action of N is 
already transitive on S'\ {1+}), so that G is transitive on Oj, for j = 1, 2, 3. 
As base-point in O3 (similar choices can be made for Oi and O2) choose 
(1+, 1+, 1"). The stabilizer of (1+, 1+, 1~) is the subgroup MA. 

• Let xi,X2,X3 be in Oq. We may assume w.l.o.g. that xi = , X2 = 1~ 
and X3 ^ 1^. The stabilizer of (1+,1~) in G is MA. The orthogonal 
projection of X3 on the hyperplane orthogonal to (l"*", 1~) is not and can 
be mapped by M to ce2, with < c < 1, so that there exists t G R such 

that x^ is conjugate under M to the point (tanht, , 0, . . . , 0) = at{e2). 

cosht 

Hence any triplet in Oq is conjugate under G to the triplet (1+, 1^, 62). Its 
stabilizer in G is the compact subgroup {k € M, ke2 = 62} — SO{n — 2). □ 

When n = 2 (5 is the unit circle), then there are two open orbits in 
S X S X S unde the action of SOq{1,2), each characterized by the value 
of the orientation index of the three points in S. It is possible to remedy 
to this fact by using the slightly larger (no longer connected) group 0(1,2) 
instead of SOo{l,2). 

Let da be the Lebesgue measure on S and let Un-i = jg da{x). Also set 
p = Under the action of G, the measure is da is transformed according 
to 

/ f{g-\x))da{x)= [ f{y)K{g,yffda{y) (11) 
JS JS 
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Let A be in C. For / in C°°{S), the formula 

7rx{g)fix) = 4g-\xr+^ fig-\x)) (12) 

defines a representation of the group G, which wih be denoted by ttx. It is a 
continous representation when the space C°°{S) is equipped with its natural 
Frechet topology (see [T^ for a systematic study of these representations). 
The representations vr^ and tt^x are dual in the sense that, for all (fjip G 

/ T^-\{g)'fis)TTxig)ipis)ds = / (p{s)'4'{s)ds (13) 
Js Js 

as can be deduced from the change of variable formula (jlip. For A pure 
imaginary, the representation tta can be extended continuously to L'^{S) to 
yield a unitary representation of G (this is the reason for using p + A in the 
definition of vr;^). Observe that the action of K is independant of A and the 
constant fonction Ig (equal to 1 evereywhere) is fixed by the action of K. 
For Q in C, let ka be the kernel on S" x 5 defined by 

ka{x,y) = \x- ^1"''+". 

It satisfies the following transformation property under the action of an 
element g of G: 

ka{g{x),g{y)) = K{g,x)~2+^ K{x,y) K{g,y)~^+^ (14) 
for all X, y in S. 



2 Construction of an invariant trilinear form 
2.1 Formal construction 

Let Ai,A2,A3 be three complex numbers. Let T be a continuous trilinear 
form from C°°{S) x C°°(S') x C°°(S') into C. The functional T is said to be 
invariant w.r.t. vta^ , vtaj , vtaj if, for every /i,/2,/3 in C°°(5), and every g in 
G, 

T{7rxi{g)fi,T^X2{g)f2,T^X3{g)h) =T{fi,f2,h) ■ (15) 

By Schwartz's kernel theorem, there exists a unique distribution T on 
S X S X S, such that 

nh,f2,h) = T{h®h®h) (16) 
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where, as usual, fi® f2® fs is the function on S x S x S defined by 

(/l ® /2 O /3)(2;i,X2,X3) = /i(xi)/2(x2)/3(a;3). 

Let ai,a2, as be complex numbers, and set a = (ai, 02, 03). Let be 
the kernel on S x S x S defined by 

Ka{xi,X2,X3) = kai{x2,X3) ka^ixs^Xi) ka3{xi,X2) (17) 

For /i, /2, /a three functions in C°°{S), define the trilinear functional /C^ 

by 

^a(/i,/2,/3) = / Kaixi,X2,X3)fi{xi)f2ix2)f3ix3)da{xi)da{x2)da{x3) 
JSxSxS 

(18) 

whenever it makes sense. 

Theorem 2.1. Let A = (Ai,A2,A3) G C^. Define a = (01,02,03) by the 
relations 

01 = — Ai + A2 + A3 

02 = -A2 + A3 + Ai (19) 

03 = —A3 + Ai + A2 . 

Then 

K.a{T^x^{g)fi,Tix^{g)f2,'^\-M)h) = ^a{h, f2, h) , (20) 
whenever the integral on the right handside is defined. 

The proof is obtained by the change of variables yj = g~^ (xj) {j = 1, 2, 3) 
in the integral defining the left hand-side, using ()14p and (jlip . Observe that 
the right-handside integral converges if, for i ^ j, the supports of fi and fj 
are disjoint, or if /^xsxs l-^ct(^i; ^2, 2:3)1 da{xi)da{x2)da{x3) < +00. 

The version of the trilinear functional in the noncompact picture will be 
useful. Let a = (01,02, "3) £ and set for yi, 2/2,^3 £ M"~^ 

Myi,y2,y3) = |yi - y2r''+"^|2/2 - y3r''+"My3 - ^11""+"^ (21) 

and, whenever it makes sense, let be the associated distribution on 

Jaiv) = j Jc.{yi,y2,y3)f{yi,y2,y3)dyidy2dy3 , (22) 

(93 G C°°(M""^ X X M"-i)). Moreover, let l-a be the function defined 

on M"'i X M"-i X M"-i by 

^a(yi,y2,y3) = (l+|yi|^)~''"^"^(l+|y2|^)~''~'^'^(l+|y3p)~''"^"^ • 



8 



Proposition 2.1. Let f G C°°(S' x S x S). Then 

/Co(/) = 23("-i)j-,((/oc)vI/,) (23) 
whenever the left hand side is defined. 

This is merely the change of variable (jlOp in the integral (jlSp . 

2.2 Integrability of the kernel Ka 

Theorem 2.2. The kernel Ka is integrahle on S x S x S if and only if 

^a,>-p, i = 1,2,3 (24) 
'Rai + Ka2 + Jftas > -p (25) 

Proof. It is enough to study the integral when the a's are real, in which case 
the kernel Ka is positive. Let W be a (small) neighborhood of (l"*", 1^, 1^) 
in S X S X S. Let g be in G. As x varies in S, j{g,x) remains bounded 
from below and from above. Thanks to the transformation property of the 
kernel ka (|14p . the integrals of over U and over g(pl) are of the same 
nature. As U meets all G-orbits, a partition of unity argument shows that 
the integrability over S of the kernel is equivalent to its integrability 
over U. Now use the stereographic projection to see that the integrability 
of Ka over U is equivalent to the integrability of Jq. over c^^{U), which is a 
(small) neighborhood of (0,0,0) in M""^ x R""^ x M"-^ 

Hence, we are reduced to study the convergence of the integral 

I6I«5 

where 5 is a small positive number. Set 

yi = 6, 2/2 = 6- 6, 2/3 = -6 + 6 • 

Then, after integrating with respect to yi, the integral Ii is seen to be of 
the same nature as the integral 

h = j^^^^, |2/2r^-^|2/3r^-l2/2 - ysr-" dy^ dys . 
|y3|<<5 

Let S be the unit ball in M(""1) x and let da be the Lebesgue 

measure on S. Use polar coordinates (r, ((T2,<T3)) defined by 

y2 = r(T2, ys = ras, = jyal^ + |2/3p, kip + Icrgl^ = 1 



9 



to obtain that I2 is of the same nature as 

JT. 
The first factor converges if and only if condition (|25p is satisfied. It remains 
to discuss the convergence of 



Let 5 > and consider the following open subsets of S : 

S2 = {(0-2, (T3), |fT2| < 5}, Sa = {((72, 0-3), IctsI < 5}, Si = {(0-2, 0-3), |cT2-a3| < 

For 5 small enough, these sets are disjoint (recall that |(T2p + |o"3p = 1). 
On S \ (El U E2 U S3) the function to be integrated is bounded from below. 
Hence the integral 14 is convergent if and only the integrals 

Kj= I |a2|"«-''k3|"^"1fT2-a3|"^~^dcr 

are convergent for j = 1, 2, 3. Let j = 2. On S2, both \(t^\ and \(T2 — o"3| are 
bounded from below, so that it is equivalent to study the convergence of the 
integral 

/ |a2|°3-^(icr . 
Jt.2 

We are reduced to a classical situation and K2 converges if and only if 
d'i — p > —{n — 1). A similar study applies to Ki and K^. This completes 
the proof of Theorem 12.21 □ 

Corollary 2.1. Let Ai,A2,A3 he three complex numbers satisfying the con- 
dition 

< 3f?(Aj) < p . 

Define ex by the relations (jl9p . Then the kernel is integrable, and the 
corresponding trilinear form is invariant for Tr\^ , ttaj , T^x-j ■ 

Proof. The conditions on A = (Ai,A2,A3) imply that R{aj) > —p for j = 
1, 2, 3 and 3?(ai + 02 + 03) > 0. Hence a is in the domain of integrability 
oiKc. □ 

The corollary covers all interesting cases for spherical unitary series, 
provided one excludes the trivial representation. In fact, the parameter A 
for such a representation can be chosen either as pure imaginary (principal 
series) or satisfying < A < p (complementary series, excluding the trivial 
representation) . 
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2.3 Analytic continuation of /C^ 

The main result of this section concerns the analytic continuation of /Cq 
beyond its domain of integrability. 

Theorem 2.3. The map a i — > /Cq., originally defined for a in in the 

domain described by the conditions (I24p and (125p . valued in T)'[S x S x S) 
can be extended meromorphically with at most simple poles along the family 
of hyperplanes in C x C x C defined by the equations 

aj = -p-2k, j = 1,2,3, ken (26) 

ai + a2 + a3 = -p-2k. A; e N (27) 

Proof. Let / be in C°°{S x S x S) and consider the integral 

/ Kaixi,X2,X3)f{xi,X2,X3)da{xi)da{x2)da{x3) . 
JSxSxS 

to be meromorphically continued. Repeating the argument given supra dur- 
ing the discussion of the integrability of the kernel Ka , we may assume that 
/ has its support contained in a small neigborhood of the point (l"*", l"*", l"*"). 
Further, transfer the integral to the noncompact picture (cf (p3|) ). and study 
the analytic continuation of 

a^M{foc)^c.) 

Now = / o c is in C;?°(M"-^ x M"-^ x M"-i), and both a i — > and 
a I — > ^cx~^ are entire fonctions on C^, so that it is equivalent to study the 
meromorphic continuation of J^a as a distribution on M.'^~^ x x 

The kernel Jq. is invariant by translations by " diagonal vectors" . To take 
advantage of this remark, make the change of variables 



zi = yi, Z2 = yi- ys, ^3 = yi - y2 

in the integral 

Ja 

to obtain 

JM = I k2r^+"'k3r''+"1^2 - Zs\~P+'''i^{z2,Z3)dz2dzs 



= J \yi-y2\ ''"^"^1^2-2/31 ''^"^\y3-yi\ ''^"^f{yi,y2,y3)dyidy2dy3 



1-1 vlBi-l 
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where we have set 

i>{z2,Z^) = / <p{zi,Zi - 23, Zi - Z2)dzi . 

Now observe that ^|J is in C^(M."'~^ x M""^). Hence we arc reduced to 
studying the analytic continuation of the distribution on W^"^ x M""^ 
defined by 

= Ia{z2,Z2,)i){z2,Zz)dZ2dzz (28) 

for il) in C^(M''-i x R"-^), where we set 

Icc{z2,Z^) = \Z2\-'^''^Z,,\-P+^'\Z2 - ^31"^+"^ • (29) 

For (5 > 0, consider the following open subsets of R""^ x R""^ 

^0 = {(^2,23), \Z2\ < 5, \zz\ < 5, \Z2 - Z3\ < 6} 

s s s 

Ui = {(2:2,2:3), \z2\ > 2> ^3! > 2' 1^2 - 2:31 < 2} 
^2 = 1(22,23), I22I < 2' pal > 2' 1^3 - 22I > -| 
W3 = |(2;2,2;3), I22I > -, I23I < -, 12:2 - 23I > -| 

s s s 

Woo = {(2:2,23), |2;2| > -, |2;3| > -, I22 - 2:3! > -} . 

The family of these five open sets form a covering of R"~^ x R"~^. Let study 

the restriction of the distribution Xq, to each of these five open subsets. 

If Suppltp) c Uoo, there the Xq(V') extends as an entire function, because 
loi has no singularity on Uoo- Next assume that Supp{'ip) C U2- Set 

M^2) = [ \z3\-''+'''\z2 - Z3\-P+'''1P{Z2, Z3)dZ3 , 

so that 

= [ |22r^+"^0a(22)d22 . (30) 

As 1 23 1 and \z2 — 23 1 are bounded from below on U2, the function (pa is 
in C^(R"~^) and a 1 — > (p^ is an entire function on . On R"~^, the 
distribution- valued function s 1 — > I22I* extends meromorphically on C, with 
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simple poles at s = —{n — 1) — 2k, A; G N (see [6]), so that the integral (pO]) 
extends meromorphically to with simple poles along the hyperplanes 
a2 = —p — 2k. A similar analysis can be done over Ui and U^. 

To sum up what we have already proved, introduce the family Ai of 
meromorphic functions on having at most simple poles along the hy- 
perplanes {oj = —p — 2k}, j = 1,2,3, k £ N. Notice that they are the 
hyperplanes corresponding to the conditions ()26p . 

Proposition 2.2. Letij^ be inC^{W^~^ xW"'"^) and assume that Supp{ip) ^ 
(0, 0) . Then the function a i— t- belongs to the class M. 

Proof. In fact, choose 6 small enough so that Supp{ip) CiUq = 9. Use a 
partition of unity to write '0 as 

where ipj £ C^°°(IR"-i x ]R"-i) and Supp{ipj) C Uj,j = l,2,3,oo. The 
previous analysis shows that a i — > Za{tpj) extends meromorphically on 
with at most simple poles along the hyperplanes aj = —p — 2k, /c G N for 
j = 1, 2, 3, whereas Xa{ipoo) is an entire function of ex. □ 

Now we use a priori the existence of the meromorphic continuation of 
such integrals (see p^)- Moreover, the poles are located on a locally finite 
family of afhne hyperplanes (of a rather specific type, but we won't need this 
result). Let % be such an hyperplane (to be determined), but not included 
in the family of hyperplanes given by conditions (j26p . Let o;" = [a\, a2,ot^) 
be a regular point in %, (i.e. not contained in any other hyperplane of poles). 
The Laurent coefficients at cx^ are distributions on M"-i X R"-\ and, by 
Lemma 12.21 their supports have to be contained in {0,0}, hence they are 
derivatives of the Dirac measure (^(o.o)- So, if Xq, does have a pole at a", 
there exists a smooth function p on W^^"^ x IR'"'^-'^ with compact support and 
identically equal to 1 in a neigbourhood of (0,0), and a polynomial P on 
M"-"-*^ X M"^^, homogeneous of degree k such that Ia{pP) does not extend 
holomorphically at o;^. For t G R*, let ptiz'z^zz) = p{tz2,tz^). Use the 
change of variables {z2 ^ tz2, z^ i— tz^) to get □ , for a / oq 

Zo,{ptP) = I \z2\-P+''-'\z^\-P+''^\z2-Z^\~P+''^ p{tZ2,tz^)P{z2,Z^)dZ2dz^ 

= |t|-"3-"2-°i-pr*^Xc.(p^') • 

"^Following a traditional way, we write Ta.{'^) as an integral. What is really used here 
is merely the homogeneity of • 
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Hence 

(1 _ \t\-^^-<-^-^^-Pt-^)l^{pP) =l^{{p-pt)P) . (31) 

Now observe that the support of (p— pt)P does not contain (0, 0), and hence, 
a I — > Ia[{p — pt)P) belongs to A4. The assumption that Za{pP) does have 
a pole at Qq forces the condition 

VtGlR* 1 - |t|-"3-°§-"?-Pt-^ = . 

In turn, this condition amounts to 

A; G 2N, al + al + al = -p-k . 

Moreover, (j3ip shows that has at most a simple pole along the corre- 
sponding hyperplane. This achieves the proof of Theorem 12.31 □ 

As the invariance condition remains true by analytic continuation. The- 
orem [2]3] can be reformulated for trilinear invariant functionals (cf Theorem 

EH). 

Theorem 2.4. Let A = (Ai, A2, A3) in C^. Assume that 

-Ai + A2 + A3 ^ -p - 2N 
Ai-A2 + A3^-P-2N 
Ai + A2 - A3 ^ -P - 2N 
Ai + A2 + A3 ^ -P - 2N 

Set OL = {ai,a2-,Q.^) where (cf (fT9]) ) 

ai = — Ai -I- A2 -I- A3 

02 = — A2 + A3 -I- Ai 

03 = —A3 + Ai -I- A2 . 

Then (/i, /2, /s) 1 — > Tx{fi, /2, /s) = K,a{fi ® f2® h) is a well defined non 
trivial trilinear invariant functional w. r. t. the representations {'Kx^ , ttaj j t^x^ ) • 

The next result was obtained some time ago by the present authors, and 
has been generalized to other geometric situations in a collaboration with 
T. Kobayashi and M. Pevzner (see [3]). To state the result, consider the 
evaluation of the functional /Cq. against the function (8) (8) I5, where I5 
is the function which is identically 1 on S. Let 
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I{X) = K{cx) = / / / kaA^2,X3)ka2{x3,xi)ka.i{xi,X2)da{xi)da{x2)da{x3) 
Js Js Js 



where a and A are related by the relations (jl9p . 

Proposition 2.3. Let X = (Ai, A2, A3) in and assume that the conditions 
([32]) are satisfied. Then 



3(n-l) 

\ ^ ^ (33) 

;+A3+p -jp|- — A1+A2+A3+P Ai-A2+A3+p -jp|- Ai+A2-A3+p -| 



r(/5 + Ai)r(p + A2)r(p + A3) 

Remark Both sides of the formula are meromorphic functions on C^, and 
they are equal where defined. Notice that /(A) = K{ol) has simple poles 
exactly as prescribed by Theorem 12.31 

This result allows to strengthen the previous theorem. Define 



^ 

Ai+A2+A3+p -jp^ -Ai+A2+A3+p -jp|- Ai-A2+A3+p -j-p^ A1+A2— A3+p -^ 

Theorem 2.5. The distribution-valued function a 1— /Cq, extends as an 
entire holomorphic function on C^. The trilinear functional Tx defined by 

fx{flj2,f3)=JCMl®f2^f3) 

on C°°{S)xC°°{S)xC°°{S)) is invariant with respect to {ttx^ , ttx^ , ttx.^), where 
(Ai,A2,A3) are related to (01,02,03) by the relations (|19p . The trilinear 
form Tx is not identically provided the two following conditions are not 
simultaneously realized 

• 3i, 1 < j < 3, XjG-p-N 

• (at least) one of the conditions (132p is satisfied . 

Proof. The function cx i — > fCa extends holomorphically near any regular 
point of the hyperplanes defined by conditions ()26p and (j27p . so is holmo- 
morphic outside of the set where ar least two hyperplanes of poles meet. But 
this set is of codimension 2 and hence a 1 — > /Cq extends as a holomorphic 
function to all of C^. If none of the conditions (132p is satisfied, then /Cq, is 
a multiple (7^ 0) of fCa which is different from on VL. If Aj ^ —p + N for 
j = 1, 2, 3, then /(A) 7^ and hence /Cq. is not identically 0. □ 



15 



3 Uniqueness of the invariant trilinear form 

3.1 Induced representations and line bundles 

For this part, it is useful to realize the representation ttx as acting on smooth 
sections of a line bundle over S. 

Recall that the stabilizer of 1+ in G is the parabolic subgroup P = 
MAN. The left invariant Haar measure on P is 

f{p)dp = / / f {man)dmdadn . (34) 
p JmJaJn 

For A G C, denot by xx the character of P defined by 

Xximatn) = e*'^ . 

The modular function of P is given by 6p{man) = e~^''('°^") = X-2p{p) (see 
e.g. |17j Lemma 5.5.1.1), so that, for any q in P 

J^fipq)dp = X2p{q) J^f{p)dp (35) 

Let £x be the space of functions / in C°°{G) which, for all g in G, p in 
P satisfy 

f{9P)=X-{p+\){p)f{9) (36) 

Then G acts on £x by 

^x{g)f{i) = f{g-^i) (37) 

To any function / in associate the function / defined on S by the 
formula 

f{s) = f{k) (38) 

where k is any element in K satisfying kl^ = s. As f{km) = f{m) for any 
m in M, the right handside of ()38p does not depend on m, but merely on 
kl+ = s. 

Now, let g be in G, let s be in S, and let khein K such that kl^ = s £ S. 

Let 

g~^k = k{g^^k) a{g~^k) n 
be the Iwasawa decomposition of g^^k. Then 

k{g-^k)l+ = {g-^k){l+) = g-\s) 

and 

K{g'\s) = K{g-^k, 1+) = X-iHg-^k)) . 
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Hence, 

{U,{g)f)~is) = f{g~^k) = fiHg-'k) a{g-^k) n) = Kig~\sr+^ fig-Hs)) 

so that f ^ f is an intertwining operator for 11^ and vta- 

Let X be a character of A. Denote by the representation of P on C 
given by 

man.z = x{^)^ ■ 

Form the hne bundle L = = G x p over S, and let C = be the 
space of sections of L. Then G acts naturally on C by left translations. 
As sections of L can be identified with functions on G transforming by 
under the right action of P, the representation of G on £ is equivalent to 
TTA if X = Xp+X- 

Now take three characters xi = Xp+Xi,X2 = Xp+A2,X3 = Xp+Ag of A, set 
A = (Ai, A2, A3) and form the "exterior" product bundle Lx = i^i ^ ^X2 ^ 
as a line bundle over S x S x S. Let Cx be the space of C°° sections 
of this bundle. Let the group G acts naturally on this space by diagonal 
action. Then a trilinear invariant functional on C°°{S) x C°°{S) x C°°{S) for 
ttAi ) 7'"A2 ) '''"A3 corresponds to an invariant linear form on Cx- 

In turn, such an invariant linear functional on Cx can be viewed as an 
invariant distribution density for the dual bundle L*^ (see Appendix). 

The main tool to study these invariant distributions is Bruhat's theory, 
which is presented in the Appendix. We use heavily the description of orbits 
of G in S" X S" X S" (cf Proposition O]). 

Theorem 3.1. Let X = (Ai,A2,A3) be three complex numbers, let a = 
(01,02,03) be defined as in (I19p and assume that they satisfy the following 
generic conditions 

i) aj ^ -p -2n for j = 1,2,3 

ii) oi + 02 + 03 ^ — /9 — 2N. 

Then any trilinear invariant form for t^x^ , t^x^ , ttx^ is proportional to the 
form Tx- 

Proof. Denote by T = Tx the distribution density on S x S x S for the 
bundle L*^ corresponding to an invariant trilinear form for ttx-^ , ttaj , t^x^ ■ 

Step 1 : contribution of Oq 

We use freely of the notation presented in the appendix. Consider the re- 
striction T' of T to the open orbit Oq. Then Oq is a homogeneous space 
under G, the stabilizer H of the base point (l"*", 1~, 62) is compact. There is 
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an invariant measure on G/H, the group H acts trivially on the fiber of L\, 
so there exists exactly one (up to constant) invariant distribution given by a 
smooth density. But we already know that Ka has the right transformation 
property. Hence on Oq, after multiplication by a constant, we may assume 
that T' coincides with the restriction of /C„ to Oq. But the assumptions 
on A guarantee that /C^ can be extended (by analytic continuation) as an 
invariant distribution on S x S x S. Hence, in order to prove that T is a 
multiple of /Cq, (i.e. to prove the uniqueness statement) we need only to 
prove that an invariant distribution which vanishes on Oq is identically 0. 
In other words, we may (and hence do) assume that 

Supp{T) C Ui<j<40j. 

and proceed further to prove that T = 0. 

Let us mention that the argument given here should be modified for the 
case of the circle (i.e. as n = 2), because there are two orbits for 500(1)2) 
in S X S X S. To restore uniqueness, one can also consider the full group 
0(1, 2) instead. The rest of the proof is unchanged. 

Step 2 : contribution of Oi, (92,03 
We now show that 

Supp{T) n O3 = . 

Observe that for 1 < j < 3 

o~ = u O4 

so that (Oi U O2 U O4) is a closed subset oi S x S x S. Let 

X = S X S X S\{OiU02U04) . 

Then X is an open submanifold oi S x S x S, acted by G, and O3 is a closed 
orbit of G in X. Let T" be the restriction of T to X, so that Supp{T") C O3. 
Now apply Bruhat's theory. 

The normal space at the base point (!"*", 1"*", 1~) is identified (via the 
Riemannian metric on S* x S* x S*) with 

iVo = {(^,-e,o),CGri5}~M"-i 

The stabilizer of of the base point is H = AM, and there is an invariant 

measure on O3, so that = 1- The group M acts on Nq by its natural 
action. As M modules, Nq and its dual are equivalent, so that the spaces 
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S^{Nq) and VkiNo) are equivalent M-modules. The space of M-invariants 
in VkiNo) is if /c is odd (and one-dimensional, generated by |^|'^ if k is 
even, but we won't need this fact). On the other hand, an element at of A 
acts on A^'o by dilation by e~*, so that it acts on S''{Nq) by multiplication 
by e~^^. The element Oj acts on the fiber Lq of -Z^xi.X2,X3 (-'-^' -'-") 
g{p+Ai+p+A2-p~A3)t_ Hence the element at acts on S''{No) L*q by multipli- 
cation by 

g(-fc-p-(Ai+A2-A3))t_ 

The assumptions on A (more precisely 03 = Ai + A2 — A3 ^ —p — 2N) 
guarantee that 

for any A; G N. Hence, by Corollary A[T1 there is no non-trivial invariant 
distribution supported in O3. 

Repeating the argument for O2 and Oi, we may (and hence do) assume 
now that 

Supp{T) C 04. 
Third step : contribution of C4 

Here we take X = x x S*, as 04 is closed. The stabilizer of the base-point 
(l"*", l"*", l"*") is P = MAN. The character xo is given by 

Xoimatn) = e"^^* . 

The normal space at (l"*", 1+, 1"*") can be identified with 

A^o = {(ei,6,e3),Ci GTiS, 6+^2 + ^3 = o}~M""ie M"-i . 

The group M acts on Nq by its natural action on each factor. Again, as 
M modules, the space S''{Nq) is isomorphic to Vk{N(i). The algebra of 
S0{n — 1) invariant polynomials on M"^^©M"^^ is generated (as an algebra) 
by ICP, Hence S''{No)^' = {0} if k is odd. On the other hand, 

an element at of A acts on Nq by multiplication by e^*, hence on SkiNo) by 
multiplication by e"'^*. It acts on the fiber Lq by e(''+''*i+''+'^2+p+A3)t_ Hence 
at acts on S^{No) C^-i by 

g(^fc-p-(AiA2+A3))i _ 

The assumptions on A (namely Ai + A2 + A3 ^ —p — 2N) guarantee that 

{Sk{No)0L*0C^-if = {0} 

for any A; G N. Hence, by Corollary A[T1 there is no non-trivial invariant 
distribution supported in O4. The uniqueness statement follows. □ 



19 



4 Another construction of a trilinear invariant form 



In this section we present a different construction of the invariant trilinear 
form 7a. Trihnear forms are connected with tensor products of representa- 
tions. Roughly speaking, a trilinear form on 1-Li x %2 x ^3 can be realized as 
an invariant bilinear pairing between T-Li ®%2 and Although this point 
of view breaks the (sort of) symmetry between the three factors, it produces 
interesting relations with questions about tensor products of representations 
(cf which was the main source of inspiration for this section). Our pre- 
sentation of the construction is formal and we don't work out the estimates 
and analytical aspects of the construction, which would follow along similar 
lines as in previous sections. 



4.1 The space S\ as a symmetric space 

Recah that = {{x,y) e S x S,x ^ y} is the orbit of (1+, 1" 
The stabilizer in G of (1^, 1^) is the subgroup H = MA. 
/-I ... 0\ 
0-10 ... 
1 



under G. 



Let J 



Then = J ^ = J. The map 



\ ... 1/ 

9 I — > JgJ 

is an involutive automorphism of G. The set of fixed points of this involution 
is the subgroup H = H U H~ , 

C / cosh t — sinh t 0^ 
= \ h = i sinht -cosht o), t e R,k e 0{n - l),detk = -1 



\ 



So H is the neutral component of the fixed points of an involutive auto- 
morphism of G. In other words, Sj can be realized as the symmetric space 
G/H via 

G/H3g^{g{l+),g{l')) . 

There exists a unique (up to a positive real number) G-invariant measure 
on 5y, namely 



G/H 



md,i.) = JJjis,t)^-^^. 



(39) 
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Another description of G/H is as follows. Let X be the set of all 2- 
dimensional oriented subspaces 11 in R-^'" such that the restriction of q to 
n is of signature (1,1). The space X is an open set in the Grassmannian 
G{2,n + 1) of oriented 2-dimensional subspaces in M^'". The group G oper- 
ates transitively onX. To the couple (s, t) in Sy, associate the 2-dimensional 
space n(s, t) = Rs©Rt, with the orientation for which (s, t) is a direct basis. 

Proposition 4.1. The mapping 11 is a diffeomorphism from onto X. 

Proof. Let s,t be in S^- Then Il{s,t) cannot be totally isotropic for q, 
because the maximally isotropic subspaces are of dimension 1. As n(,s,t) 
contains two independant isotropic vectors (s and t), the signature of q on 
Il{s,t) has to be (1,1). Hence n(s, i) belongs to X. On the other hand, 
let n be in A'. Then the isotropic cone in H decomposes as the union of 
two distinct isotropic lines, say DU A. The line D (resp. A) intersects the 
hyperplane {xq = 1} at a point s (resp. t) with s (resp. t) in S. Permuting 
D and A if needed, we may assume that the basis (s, t) corresponds to the 
orientation of 11. Then clearly n(s, t) = H. Thus 11 is a 1— 1 correspondance. 
The fact that it is a diffeomorphism is standard and left to the reader. □ 

4.2 An equivariant realization of tt^ ® iTr 

The group H is connected, isomorphic to M x SO{n — 1). For ^ any complex 
number, the function z/^ on H defined by 



{t eR,k e M) is a, character of H. 

Form the line bundle = G xj/ over G/H. A section of can be 
viewed as a C°° function F : G — > C which satisfies 



Denote by iZ,^ the space of smooth sections of over G/H. The natural 
action of G acts on by left translation is denoted by : 



When ^ is pure imaginary, the character z/^ is unitary. If F is in >C^, then 
\F{gh)\ = \F{g)\ for any h in H. Hence the expression 




F{gh) = u^{h)-'F{g) . 



(40) 




G/H 



(41) 
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is well-defined (maybe +00) and is finite if (say) F has compact support 
modulo H. The representation extends as a unitary representation for 
this inner product. 

Let a, T be two complex numbers, and consider the representations vTo- 
and TTr- As C°°{S) is a Frechet space, the projective and inductive topologies 
on the tensor product C{S) C{S) coincide, and the (completed) tensor 
product is realized as C°°(S' x S). Hence the tensor product tTo- (8) tTt is 
naturally realized on C°°{S x S). Explicitly, 

^5 7rr{g)f{xi,X2) = K{g'\xi)P+''K{g~\x2Y+^f{g-\xi),g-\x2)) (42) 

for 5 G G,/ G C^{S X 5),xi,X2 G S. 

For / in C°°{S x S), let Po-.t/ be the function on G defined by 

{P.,rf){g) = K{g,l+Y+'^K{g,l-Y+^ f[g{l+),g{l-)) (43) 

Proposition 4.2. Pa^rf satisfies the relation 

{Pa,rf){gh) = U^_r{h)-\Pa,rf){g), (44) 

for g in G and h in H . 

Proof. Recall that the elements of H fix both 1+ and 1~. Moreover, 

K{at,l^) = e^\ tGR. (45) 

Now (jHI follows □ 

The map P^^t will be regarded as a map from G°°{S x S) into Ca-r- 
Let cr and r be pure imaginary. Observe that 

\g{l+) - g{l-)\ =2K{g,l+)"2K{g,l-)'2 , 

so that 

= 2-2("-i) / |.-tp(-^)|/(.,t)p , "^'Jl ^. =2-^(»-^)||/f 

and Pa,T extends as an isometry (up to a constant) from L'^{S x S) onto the 
space of square-integrable elements of Ca-r for the inner product associated 
to gl]). 

Proposition 4.3. Po-,t intertwines the representation tTo- vr,- and i/ie rep- 
resentation la-T, i-S- for any g in G 

Pa,T O (vr<x Trr){g) = la-rig) ° P^,r (46) 
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Proof. Let / be in C°°(5 x S). For 7 in G, 

K{g, l-r-'^Ki^-\gil+)r+"Ki^-\g{l-)y+^ . . . 

On the other hand, 

Ia-r{l)Pa,rf{9) = 

K{j-'g,i+r+^KiT'9,i-r^^f{T\gii+),g-Hi-))) ■ 

The two expressions are easily seen to be equal by using ([6|). □ 

4.3 Construction of a (H, z/^) covariant function for nx 

The next step consists in finding in the representation space for vr;^ an ele- 
ment Oa,^ which transforms under the action of H by the character z^^. In 
general, it will be a distribution on S (a commun fact in harmonic analy- 
sis on semisimple symmetric space (cf [15]). We use a geometric approach 
through the realization of G/H as X (see Proposition 14. ip . 

Let n be an element of X and s be in S. Observe that is a an 
(n — l)-dimensional space, complementary to H, and the restriction of q to 

is negative-definite. Define 

^(n,s) = 2(-(?(projnx5))^ . (47) 
Proposition 4.4. Let (si,S2) be in S'^. Then, for any S3 in S, 

^msuS2), ss) = 2^1^*1^ . (48) 

Proof. Let as = projn^ ^3. Then there exist real numbers ai and 02 such 
that o"3 = ?3 — aisi — 0282- The numbers ai and 02 are determined by the 
conditions 

[0-3, si] = [0-3,52] = . 

Hence, 

1 - S3.S2 - ai(l - S1.S2) = [53,52] - ai[si,S2] = [0-3 + "252, S2] = 
so that 

"1 = 1 = I r 

1 - S1.S2 V Si - S2 
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and similarly 

I-S3-S1 /|S1-S3|\2 

Now 

-5(0"3) = -[S3 - "1^1 - 0252, S3 - "1^1 - 0252] 

= 2ai[si,S3] + 2a2 [52,53] - 2qiQ2[si, S2]) 
|si - S2I 



_ / |5l - 53||g2 - 53| \^ 



□ 



Proposition 4.5. Let g he in G. Then, for any H in X and s in S, 

^{g{n),g{s))=K{g,s)^{'K,s) (49) 

Proof. As g preserves the form q and hence the orthogonality relative to 
for any 11 in 

^oprojnx =projg(n)-L0 5 • 

Hence 

<l{projg{ji)±g{s)) = {g^o'^q{projg(ji)^g^ = (5 s)o)"^g(projnx s) 
so that 

^{g{Tl),g{s)) = {g7)^'^{Ii,s) = K{g,s)^{'K,s) 

□ 

Let Ho = n(l+, 1~) = {x = (xo,xi,0, . . . ,0),xo,a;i G M}, and let 

*o(s) = ^'(no,5) = |l+-s||l--s|=2(xi + --- + x2)l . (50) 



As a consequence of the previous proposition, the function ^'o has a nice 
transformation law under the action of H. 

(coshi sinhi 0\ 
sinh t cosh t 1 , where t is in R, and k is in 
OA;/ 

SO{n — 1). For any s in S 

^o{h{s)) = K{h,s)^o{s) . (51) 
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Let A and ( be two complex numbers. For s in 5 define 



QxAs) = Ms) 



-p-X 



|1+_S|-P-A+C|1- 



-p-x-C 



1 -s 

Proposition 4.6. The function ©A,f satisfies 

for all h in H. 

Proof. Let h be in H. Then 

^A(/i)©A,c(s) = ^^ih-\sr+^Qx,dh-\s)) 

1+ -/i-i(s) C 



(52) 



(53) 



K{h~\sr+^ ^oih-\s)r^'^ 



^a(s)-P-^ (e2*)l 



p~X 


K{h 




c 

2 


1+-S 










1 - s 



M^)&x,ds) . 



by using (jl5|) . 



□ 



4.4 The duality between J,^ and tta 

Now define the corresponding Fourier transform J~x^( '■ for F a smooth 
section of with compact support modulo H, define ^c,xF by 



Fx,^F{s)= [ F{g)7Tx{g)ex,c{s)dfi{gH) 

JG/H 



(54) 



Observe that, thanks to ()53p the integrand is a function on G/ so that 
the integral makes sense, the result being in general a distribution on S. 



Proposition 4.7. For any g in G 

^x^ckia) = T^x{9)^x,(, 



(55) 
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Proof. Set = Oa,^, = J^x,(, let F be in £f with compact support modulo 
H and let 7 be in G. Then 

J-o/^(7)F(s) == / F(7-M eig-Hs))dfi{gH) . 

Jg/h 

Set 7~^(7 = / and use the invariance of the measure d/i to obtain 

^o/^(7)F(s) = / F{l)K{l-^r\sr+'eil-Hl-\s))df,m 
Jg/h 

1 .\p+^ [ -.r^i.WP+^f^d-^l^-^i 



Jg/h 

Jg/h 

= Kir\sr^'TF{r'is)) = {n^{j)TF){s) . 



4.5 Application to trilinear forms 



□ 



Theorem 4.1. Let Ai, A2, A3 be three complex numbers. For /i,/2,/3 three 
functions in C°°{S), let T{fi, f2, fs) be defined by 



A3, Ai— A2 Px,M{h®h),h)- (56) 

Then, for any g G G, 

T{7rx^{g)fi,irx2{g)f2,T^X3{9)f3) = T'C/i, /2, /s) , 
whenever the right hand side is well defined. 

Proof. Let be in G and let /i, /2, /s be three functions in C°°{S). Then 

T{TT\^{g) fi,TTx.,{g) f2,T^x-i{9)h) = (y^~x-iM~Mf>^iMi'^>^ii9)fi®T^X2i9)f2),T^X3 

= {F-x-iM-X2h^-X2{9)Px^M{h «> f2),T^Xi{9)fz) 

(use (06])) 

= (7r-A3(5')(-^-A3,Ai-A2-PAi,A2(/l ® /2)),vrA3(5)/3^ 

(use dSSl)) 

= n/i,/2,/3) . 

(use (USD). □ 
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Making explicit the right handside of (j56p shows that it coincides (up 
to a constant) with the former expression of T\ where A = (Ai, A2, A3) (see 
Theorem El]). 

Appendix : Invariant distributions supported by a 
submanifold 

Here is a presentation of the main results in Bruhat's theory [2], but written 
in terms of vector bundles and distribution densities (in the sense of [7] ch. 
VI). We sketch the main steps of the proof, following [17j . 

A.l Distribution densities for a vector bundle over a manifold 

First recall the composition of a distribution with a diffeomorphism. Let X\ 
and X2 be two open sets of and let $ : Xi — )■ X2 be a C°° diffeomorphism. 
Then there is a unique continous linear map ^* : 'D'{X2) — )• T>'{Xi) which 
extends the composition of functions, i.e. such that = / o $ for / G 

C{X2). 

A distribution density on a manifold X is by definition a continuous 
linear form on C^{X). Let u he a distribution density on X. Let (Xk,k) 
be a local chart, i.e. k is a diffeomorphism of an open set of X onto an 
open set X^ of M^. Then the formula 

Uk{(p) = u{ipo k) (57) 

for if E C^(Xk) defines a distribution on X^, called the local expression of 
u in the chart {X^, k) . Further, let (X^, k) and (X^', k') be two overlapping 
charts, and let and u^' the corresponding local expressions of u. Let 

: k'{x^ n x^>) ^ k{x^ n x^>) 

be the change of coordinates (equal to k o k'~^). Then 

u^, = I detd$| $*nK in K'{X^r\Xf,i) . (58) 

Conversely, suppose we have an atlas of charts (X^ , k) covering the 
manifold M and suppose that for each k we are given a distribution £ 
P'(Xk). Assume further that for any two overlapping charts {Xi^,k) and 
{Xi^',k'), the condition ([58]) is satisfied. Then the system (^{uk),k G T) 
defines a unique distribution density n on X such that, for k in and 
if £ C^(Xk) condition (f57]) is satisfied. 
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The space of distribution densities on X is denoted by V'{X). A smooth 
density is a density the local expressions of which are C°° functions. The 
smooth densities are sections of a line bundle called the density bundle 
r2(X). It is very similar to the bundle of differential forms of maximal degree 
on X, in the sense that their transition functions just differ by an absolute 
value. 

This definition can be extended to the case of vector bundles. Let vr : 
L — )• X be a C°° vector bundle over M, with model fiber Eq. Let {Xi) be 
a family of open subsets of X such that over each X^ the bundle can be 
trivialized. Let : 7r~^(Xj) Xi x Eq and j : 7r~^(Xj) Xj x Eq be 
two trivializations of the bundle over two overlapping subsets Xi and Xj. 
Then the map o over {Xi f^Xj) x Lq ^ {Xi r\Xj)x Lq is of the form 

{x,v) {x,gij{x)v) 

where gij{x) is in GL{Lq) and the map {transition functions of the bundle) 

Qij : XiHXj ex<-^ gij{x) G GL{Eq) 

is C°°. A distribution density for the bundle L is a system {ui) of distribution 
densities on X^ with values in Lq such that 

Ui = 9ijUj in n Xj . 

Denote by T>'{X, L) the space of distribution densities for the bundle L. 

Let be the space of sections with compact support of the bundle 
L. Then the dual of is identified with the space 'D'{X, L*). The smooth 
elements in the dual (those given locally by integration against a smooth 
function) are the C°° sections of the bundle Q.{X) ® L* . 



A. 2 Invariant distribution : the case of a homogeneous vector 
bundle 

Let G be a Lie group acting transitively on a manifold X. Let o be a 
base-point in X, and H = G° he its stabilizer in G, so that X ~ G/H. An 
element h of H acts on X and fixes o, so that by differentiation, it acts on its 
tangent plane by (say) To{h). The tangent space TqX can be identified with 
fl/f). The element h acts on g by the adjoint action Adg(/i), and preserves 
the subalgebra f) on which it acts by Adf, h. Hence its acts on g/f), and this 
action coincides with ro(/i). This action satisfies 

detro(/^) = ^^4^ (59) 
^ ^ det(Adi,/i) ^ ^ 
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The modular function 5g of a Lie group G is defined by 
(^G(9) = |detAd(<7-^)| . 

so that 

|detro(/i)| :=xo{h) = ^f^ . (60) 

A homogeneous vector bundle L over X is a vector bundle L together 
with an action of the group G on L by bundle isomorphisms. If g is in G 
and X in X, then g maps the fiber into -^^^(x) by a linear isomorphism. 
In particular, H acts on Lq by a representation (say) r. Conversely, given a 
representation r of in a vector space E, then one constructs the bundle 
G Xr E as G X E/ ~, where ~ is the equivalence relation defined by the 
right action of H on G x E 

{g,v) {gh^^ ,T{h)v) for some hG H , 

g in G and v £ E. Any homogeneous vector bundle over X is of that sort, 
in the sense that the bundle L is isomorphic to G x^ Lq (see |16j). 

A section s : X — > L can be realized as a Lo-valued function fs on G 
which satisfies 

fs{9h) = T-\h)Mg) (61) 

and, conversely, such a function / gives raise to a section of L. Let £Jf be 
the space of smooth sections of L with compact support. The space £Jf is 
G-equivariantly isomorphic to C^{G, H, r), the space of C°° functions on G 
which satisfy (fHTl) and have compact support modulo H. The group G acts 
by left translations on £^ , and this action is equivariant with the left action 
of G on C^{G,H,t). 

The tangent bundle TX of X is an example of such a homogeneous bun- 
dle. The action of H on the fiber TqX is tq. Another important homogenous 
bundle is the bundle il.(X) of densities over X. It is a line bundle, corre- 
sponding to the character of H given by | det ((t(/i)^-'^)*) | = Xo{h)^^ (cf 
(I60p ). so that, in this context, we denote the fiber at o of the bundle n{X) 
by C -1. 

Theorem Al. There exists a non trivial invariant continous linear forms 
on £Jf if and only if there exists a non trivial linear form £^ on E such that, 
for all he H 

rih-^C = Xom ■ (62) 

More precisely, 

(jCT*f = V'{X,L*f ~ {LI » C -i)^. (63) 
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Proof. (Sketch of) An element of the dual of is a distribution density for 
the bundle L*. If it is invariant by G, then the corresponding distribution 
density turns out to be smooth. Hence we are looking for a G-invariant 
smooth section of the bundle L* ® But this is equivalent to an H 

invariant element in the fiber at o. □ 

A. 3 Invariant distribution supported in a submanifold 

Let X be a manifold and G a Lie group acting on X. Let Q be an orbit 
of G in X and assume that Q is closed. Let L be a homogeneous vector 
bundle over X. Let N be the normal bundle of Q (i.e. the quotient bundle 
TX\q/TQ). Fix a base-point o in Q, let Go = H he the stabilizer of o. Then 
H acts on the tangent space ToX, preserving the subspace TgQ, and hence 
acts on the normal space Nq = TqX/ToQ at o. 
Let T be a distribution on Q. Then the map 

C^{X)3^^{T,<f\Q) 

defines a distribution on X, which we still denote by T. 

Let be the space of smooth sections of L with compact support. 
Let T be a continuous linear form on supported in Q. Choose a local 
coordinate system on X 

{Ul, ...,Us,Vl,.. .,Vr) 

such that Vj = 0,1 < j < r are local equations for Q. The (■Ui)i<i<s form 
a coordinate system of Q near o. On the other hand let N be the normal 
bundle of Q. The family (^)i<j<r gives a local trivialization of the normal 
bundle A^. Choose a local trivialization of the bundle L, and denote by 
el,. . . ,ej the corresponding coordinates on the fiber. 

By Schwarz's local structure theorem for distributions supported in a 
vector subspace, there exists an integer A; G N (the local transversal order 
of the distribution), and for each j,l < j < I and each multidiindex a = 
(ai, . . . ,ar) with |a| = ai H + ar < k uniquely determined distributions 

on (some open subset of) C x W such that 

l<j<l \cx\<k 

Let us consider the "top terms" subcollection (T^, \ct\ = k,l < j < I). This 
collection can be interpreted as the local expression of a distribution density 
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for the bundle S^{N) L*q on Q. This is obtained by checking the way the 
collection transforms under 

• change of local trivialization of the bundle L (change the e^'s) 

• change of local trivialization of the normal bundle (change the f/s) 

• change of the local coordinate system on Q (change the w^'s). 
Denote by a^'^^T) the section of the bundle S''{N) (g) L*q associated to 

T. 

Denote by V'f^{Q,L) the space of continuous linear forms on which 
are supported on Q and of transversal order < A: on any local chart of X . 

Theorem A2. The map T i — > a^^\T) is a linear map from 2?^(Q,L) in 
V'[Q,S^{N) ®L*q), with kernel V'^_^{Q,L). Moreover, if ^ is a bundle 
diffeomorphism of the bundle L, which maps Q into Q, then 

(T(^)($*r) = $v('=)(r) , 

where denotes the action naturally induced by $ on T)'[Q,L) or of the 
restriction of^^Q on V {Q , S'' (N) ^ LJq) . 

Assume now that T is invariant by G. Obersve that invariance insures 
that the local transversal degree of T is the same for all charts of Q. 

Denote by V'^{Q,L)^ the space of G-invariant elements of T)'^{Q,L). 
The conjonction of Theorem A[T]and Theorem Al2] gives some estimate of its 
dimension. 

Theorem A3. For any A; G N, 

d\m[V'f,{Q,Lf /V't,_^{Q,Lf) < dim {Sk{No) L*^ C^^if . 

The theorem is mostly used in through the following corollary. 
Corollary Al. Assume that for any /c G N 

{Sk{No)0Ll^C^-^f = {0} . 

Then there exists no non trivial continuous G-invariant linear form on CJ^ 
such that Supp{T) C Q. 
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